Detect language in the markdown pipelines and filter on it - #86

Closes #47.

The columnar index carries CLD2 labels Common Crawl computed over the raw HTML. Those labels are unreliable for low resource languages and missing outright on a good share of rows, so "give me the Vietnamese pages" is harder than it should be.

Both markdown pipelines now run a trigram identifier over the Markdown they extracted and write the answer into two new columns, language and language_confidence. Detecting on the extracted text rather than on the HTML is the whole point: that is the text the dataset keeps. --lang keeps only one language, --min-lang-confidence sets the floor and defaults to 0.8, which is the identifier's own reliability threshold.

ccrawl content lang <url> answers the same question for a single URL and prints the text the identifier judged, which is how you find out why a page was kept or dropped.

Decisions worth arguing with

Without --lang nothing is dropped and the columns are still filled in, so an unfiltered shard stays filterable later without extracting it again.

A document with too little text to identify comes out with an empty language, and a filter drops it. A filtered export asks for documents known to be in one language, and "we could not tell" is not that.

The identifier gets a cleaned sample rather than the raw Markdown. A link heavy page, which on a news site means the homepage and every section index, carries more characters of slug than of prose, and a Vietnamese slug has no diacritics by definition. Link targets go, link text stays, because the link text is the headline. On the vnexpress.net homepage that is the difference between confidence 0.16 and confidence 1.

--lang with markdown refetch --fetch-only is a usage error rather than a silently ignored flag, since fetch-only never produces the Markdown the identifier reads. It exits 2.

User visible default change

The export schema becomes open-markdown-v3 and the default repo moves from open-index/open-markdown-v2 to open-index/open-markdown-v3, so the two schemas stay in separate repos. Anyone pinning a repo with --repo is unaffected.

Done when, checked against a real crawl

Three shards of CC-MAIN-2026-30, 62824 documents converted, --lang vie --min-lang-confidence 0.8:

markdown: 3 committed, 0 skipped, 0 failed of 3 | 647 rows | html=110.1 MB md=3.7 MB parquet=1.5 MB | 4m46s elapsed
language: --lang vie kept 647 of 62824 documents, dropped 62177 (99.0%)
language: detected eng=24266 fra=4165 deu=3915 rus=3786 spa=3388 cmn=2984 jpn=2702 por=1814

• a 200 document sample is at least 98 percent Vietnamese. 200 rows sampled by hash from the 647. 172 of them carry three or more Vietnamese function words that only exist with diacritics, which is an independent check and not the identifier's own opinion. The other 28 were read one by one: 27 Vietnamese, 1 miss. That is 199 of 200, 99.5 percent.
• run stats show the drop rate. Both lines above, plus lang_dropped and lang_counts on the journal events, because the text summary is suppressed in the machine progress modes and the drop rate is the first thing to check when a shard comes out smaller than expected.
• a v2 reader still parses v3 files for the columns they share. Unit test reads a v3 file through the old struct, and DuckDB projects the eight v2 columns out of a real shard from the run above.

The one miss is worth writing down: a mojibake page, UTF-8 served as Latin-1, whose garbled Cyrillic carries enough accented Latin characters to read as Vietnamese. 42 of the 647 rows show mojibake of some kind, most of them genuinely Vietnamese pages with broken encoding. Romanized Vietnamese written without tone marks is detected as Vietnamese, which is usually what you want.

Docs say plainly that this is a coarse pre-filter and not a substitute for a language specific classifier, in the CLI reference, the markdown reference, and the command help.

Gates

gofmt -l ., go vet ./..., go test ./..., golangci-lint run, and scripts/docs-drift.sh all clean.